Papers by André F. T. Martins
Joint Learning of Named Entity Recognition and Entity Linking (P19-2)
Copied to clipboard
| Challenge: | Named entity recognition and entity linking are two fundamentally related tasks . most approaches focus on the mention detection part, assuming the correct mentions have been detected . |
| Approach: | They perform joint learning of named entity recognition and entity linking to leverage their relatedness. |
| Outcome: | The proposed model achieves competitive results with the state-of-the-art in both NER and EL tasks. |
Revisiting Higher-Order Dependency Parsers (2020.acl-main)
Copied to clipboard
| Challenge: | Neural encoders have allowed dependency parsers to shift from higher-order structured models to simpler first-order ones, making decoding faster and still achieving better accuracy than non-neural parser. |
| Approach: | They found that neural parsers may benefit from higher-order features when employing a powerful pre-trained encoder, such as BERT. |
| Outcome: | Using a pre-trained encoder, we found that higher-order models are more accurate on full sentence parses and match of modifier lists. |
A Context-aware Framework for Translation-mediated Conversations (2026.tacl-1)
Copied to clipboard
| Challenge: | Existing systems that bridge language barriers can introduce errors leading to misunderstandings and conversation breakdown. |
| Approach: | They propose a framework to integrate contextual information into automatic translation systems . they validate the framework on customer chat and user-assistant interaction . |
| Outcome: | The proposed framework consistently produces better translations than state-of-the-art systems on two task-oriented domains. |
A Simple and Effective Approach to Automatic Post-Editing with Transfer Learning (P19-1)
Copied to clipboard
| Challenge: | Existing APE systems generate artificial triplets of source sentences, machine translation outputs and human post-edits. |
| Approach: | They propose to use human post-edits to refine black-box machine translation (MT) models by fine-tuning pre-trained BERT models on both encoder and decoder of an APE system. |
| Outcome: | The proposed method improves on a dataset of 23K sentences on x86 GPUs. |
Sparse Sequence-to-Sequence Models (P19-1)
Copied to clipboard
| Challenge: | Sequence-to-sequence models are dense and assigning nonzero probability to implausible outputs. |
| Approach: | They propose a new family of -entmax transformations that includes softmax and sparsemax as particular cases and is sparser for any > 1 . they provide fast algorithms to evaluate these transformations and their gradients, which scale well for large vocabulary sizes. |
| Outcome: | The proposed models are able to produce sparse alignments and assign nonzero probability to short list of plausible outputs, sometimes rendering beam search exact. |
SPECTRA: Sparse Structured Text Rationalization (2021.emnlp-main)
Copied to clipboard
| Challenge: | Sparse attention mechanisms are a deterministic alternative, but they lack a way to regularize rationale extraction. |
| Approach: | They propose a framework for deterministic extraction of structured explanations via constrained inference on a factor graph, forming a differentiable layer. |
| Outcome: | The proposed framework outperforms previous studies on performance and plausibility of extracted rationales. |
MLQE-PE: A Multilingual Quality Estimation and Post-Editing Dataset (2022.lrec-1)
Copied to clipboard
Marina Fomicheva, Shuo Sun, Erick Fonseca, Chrysoula Zerva, Frédéric Blain, Vishrav Chaudhary, Francisco Guzmán, Nina Lopatina, Lucia Specia, André F. T. Martins
| Challenge: | Existing datasets for machine translation quality estimation and post-editing have several shortcomings. |
| Approach: | They propose a dataset for machine translation quality estimation and automatic post-editing . they report the performance of baseline systems trained on the MLQE-PE dataset . |
| Outcome: | The proposed dataset contains human labels for up to 10,000 translations per language pair. |
Understanding the Mechanics of SPIGOT: Surrogate Gradients for Latent Structure Learning (2020.emnlp-main)
Copied to clipboard
| Challenge: | Latent structure models can mitigate the error propagation and annotation bottleneck in pipeline systems, while uncovering linguistic insights about the data. |
| Approach: | They propose a latent structure model with a pullback of the downstream learning objective. |
| Outcome: | The proposed model outperforms the known and proposed model in the same family and yields new insights for practitioners and revealing intriguing failure cases. |
Selective Attention for Context-aware Neural Machine Translation (N19-1)
Copied to clipboard
| Challenge: | Recent work in context-aware NMT considers only a few previous sentences as context . current systems fail to achieve fluent, good quality translation for a full document . |
| Approach: | They propose a top-down approach to hierarchical attention for context-aware NMT which uses sparse attention to selectively focus on relevant sentences in the document context. |
| Outcome: | The proposed approach outperforms context-agnostic baselines and context-based baselines on English-German datasets. |
Fine-Grained Reward Optimization for Machine Translation using Error Severity Mappings (2026.tacl-1)
Copied to clipboard
Miguel Moura Ramos, Tomás Almeida, Daniel Vareta, Filipe Azevedo, Sweta Agrawal, Patrick Fernandes, André F. T. Martins
| Challenge: | Reinforcement learning (RL) is an effective and robust method for training neural machine translation systems. |
| Approach: | They propose a method that leverages fine-grained, token-level quality assessments . they use a state-of-the-art quality estimation system as their token- level reward model . |
| Outcome: | The proposed approach leverages fine-grained, token-level quality assessments along with error severity levels to improve translation quality. |
Efficient Methods for Natural Language Processing: A Survey (2023.tacl-1)
Copied to clipboard
Marcos Treviso, Ji-Ung Lee, Tianchu Ji, Betty van Aken, Qingqing Cao, Manuel R. Ciosici, Michael Hassid, Kenneth Heafield, Sara Hooker, Colin Raffel, Pedro H. Martins, André F. T. Martins, Jessica Zosa Forde, Peter Milder, Edwin Simpson, Noam Slonim, Jesse Dodge, Emma Strubell, Niranjan Balasubramanian, Leon Derczynski, Iryna Gurevych, Roy Schwartz
| Challenge: | Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data, but using only scale to improve performance means resource consumption also grows. |
| Approach: | They propose to use data, time, storage, or energy to improve model performance. |
| Outcome: | The proposed methods and findings provide guidance for conducting NLP under limited resources and point towards promising research directions for developing more efficient methods. |
Latent Structure Models for Natural Language Processing (P19-4)
Copied to clipboard
| Challenge: | Latent structure models are a powerful tool for compositional data modeling and pipelines. |
| Approach: | This tutorial will cover recent advances in discrete latent structure models . it will discuss their motivation, potential, and limitations . |
| Outcome: | This tutorial will cover recent advances in discrete latent structure models . it will discuss their motivation, potential, and limitations . |
Do Context-Aware Translation Models Pay the Right Attention? (2021.acl-long)
Copied to clipboard
| Challenge: | Context-aware machine translation models fail to leverage contextual information to resolve ambiguous words and pronouns. |
| Approach: | They propose a new dataset that includes supporting context words for 14K translations that professional translators found useful for pronoun disambiguation. |
| Outcome: | The proposed model can automatically disambiguate pronouns and polysemous words when they are not in the same context. |
Towards Dynamic Computation Graphs via Sparse Latent Structure (D18-1)
Copied to clipboard
| Challenge: | Existing approaches to learn latent structure are limited by factorization assumptions or end-to-end differentiability. |
| Approach: | They propose a method that allows for end-to-end learning of latent structure predictors jointly with a downstream predictor. |
| Outcome: | The proposed method allows for unrestricted dynamic graph construction from the global latent structure while maintaining differentiability. |
Marian: Fast Neural Machine Translation in C++ (P18-4)
Copied to clipboard
Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, Alexandra Birch
| Challenge: | In this paper, we present Marian, an efficient and self-contained Neural Machine Translation framework . Marian is written in pure C++ with minimal dependencies . |
| Approach: | They present Marian, an efficient and self-contained Neural Machine Translation framework written in pure C++ with minimal dependencies. |
| Outcome: | The proposed framework achieves high training and translation speed with minimal dependencies . it is currently being deployed in multiple European projects . |
Measuring and Increasing Context Usage in Context-Aware Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Recent work in neural machine translation has demonstrated the necessity and feasibility of using inter-sentential context, but it is often not clear how much they actually utilize it at translation time. |
| Approach: | They propose a conditional cross-mutual information metric to quantify usage of context by model architectures that can use it at translation time. |
| Outcome: | The proposed method increases context usage and improves translation quality according to BLEU and COMET metrics. |
Sparse Text Generation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Current text generators require sampling from a modified softmax to avoid degenerate text . entmax sampling creates a mismatch between training and testing conditions . |
| Approach: | They propose to use entmax transformation to train and sample from a sparse language model to avoid degenerate text. |
| Outcome: | The proposed model improves fluency and consistency, fewer repetitions, and n-gram diversity closer to human text. |
Uncertainty-Aware Machine Translation Evaluation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Several neural-based metrics have been proposed to evaluate machine translation quality, but they are trained on noisy, biased and scarce human judgements. |
| Approach: | They propose a method to evaluate machine translation quality using point estimates . they combine COMET framework with Monte Carlo dropout and deep ensembles . |
| Outcome: | The proposed methods perform well across multiple language pairs and with references. |
Disentangling Uncertainty in Machine Translation Evaluation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Trainable evaluation metrics for machine translation (MT) exhibit strong correlation with human judgements, but they are often hard to interpret and might produce unreliable scores under noisy or out-of-domain data. |
| Approach: | They propose to use Monte Carlo dropout and deep ensembles to quantify uncertainty in machine translation and assess their ability to target different sources of aleatoric and epistemic uncertainty. |
| Outcome: | The proposed measures can target different sources of aleatoric and epistemic uncertainty, with a reduction in computational costs. |
Chunk-based Nearest Neighbor Machine Translation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Semi-parametric models augment generation with retrieval, but require expensive retrieval operation for every generated token. |
| Approach: | They propose a semi-parametric model which augments generation with retrieval by retrieving tokens from a datastore. |
| Outcome: | The proposed model can retrieve chunks of tokens from the datastore, instead of a single token, with a low decoding speed. |
Smoothing and Shrinking the Sparse Seq2Seq Search Space (2021.naacl-main)
Copied to clipboard
| Challenge: | entmax-based sparse sequence-to-sequence models give high scores to short hypotheses . ent max models can shrink the search space by assigning zero probability to bad hypothese . |
| Approach: | They propose entmax-based sparse sequence-to-sequence models that minimize cross-entropy and use softmax to compute local normalized probabilities over target sequences. |
| Outcome: | The proposed models remove a major source of model error for word-level tasks . the proposed models improve cross-lingual morphological inflection and machine translation . |
Sparse and Constrained Attention for Neural Machine Translation (P18-2)
Copied to clipboard
| Challenge: | Existing approaches to address coverage problem only change attention transformations . adequacy of neural machine translation is still a major concern . |
| Approach: | They propose a new approach that allocates fertilities to source words to bound attention . they propose gating architectures and adaptive attention control to control the amount of source context . |
| Outcome: | The proposed model is differentiable and sparse and is evaluated in three languages pairs. |
Jointly Extracting and Compressing Documents with Summary State Representations (N19-1)
Copied to clipboard
| Challenge: | Text summarization is an important NLP problem with a wide range of applications in data-driven industries. |
| Approach: | They propose a neural model that extracts sentences from a document and compresses them. |
| Outcome: | The proposed model generates concise and informa-tive summaries on CNN/DailyMail and Newsroom datasets and human evaluations show it outperforms existing methods. |
Adaptively Sparse Transformers (D19-1)
Copied to clipboard
| Challenge: | Recent architectures, notably the Transformer, learn powerful context-aware word representations through layered, multi-headed attention. |
| Approach: | They propose an adaptively sparse Transformer where attention heads have flexible, context-dependent sparsity patterns. |
| Outcome: | The proposed model improves interpretability and head diversity when compared to softmax-based models on machine translation datasets. |
OpenKiwi: An Open Source Framework for Quality Estimation (P19-3)
Copied to clipboard
| Challenge: | Existing open-source frameworks for QE are based on complex ensemble systems, complicated architectures, or require not well-documented pretraining and fine-tuning of some components. |
| Approach: | They introduce OpenKiwi, a Pytorch-based framework for translation quality estimation. |
| Outcome: | The proposed framework performs state-of-the-art on word-level and sentence-level tasks and is near state-outperforming on sentence- and word-based tasks. |
Scheduled Sampling for Transformers (P19-2)
Copied to clipboard
| Challenge: | Existing studies show that scheduled sampling can be applied to recurrent neural networks to avoid exposure bias. |
| Approach: | They propose to use teacher forced embeddings and model predictions to avoid exposure bias in sequence-to-sequence generation. |
| Outcome: | The proposed technique achieves performance close to a teacher-forcing baseline on two language pairs and is promising for future research. |